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FOREWORD 


This  paper  was  originally  prepared  for  presentation  at  the  Spring  Joint 
Computer  Conference,  18-20  April  1967,  at  Atlantic  City,  New  Jersey, 
sponsored  by  The  American  Federation  of  Information  Processing  Societies. 
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ABSTRACT 


This  paper  presents  Air  Force  developed  concepts  for  technical  control 
and  design  verification  of  computer  programs.  Starting  with  the  definition 
of  a  computer  as  a  deliverable  contract  end  item  requiring  a  design  and 
development  effort,  management  procedures  for  controlling  the  design  and 
development  process  are  explained.  Technical  control  of  computer  program 
design  through  periodic  design  reviews  is  outlined  and  test  concepts  for 
verification  of  computer  program  performance  are  presented.  The  techniques 
discussed  are  based  on  an  exchange  of  technical  information  between  the 
contractor  and  the  procuring  agency  at  a  series  of  discrete  milestones 
throughout  the  design  and  development  process.  The  milestones,  including 
design  reviews,  qualification  testing,  etc.,  are  described  and  the  relation¬ 
ship  of  these  milestones  to  the  design  and  development  of  a  computer-based 
system  is  illustrated.  The  techniques  are  directly  applicable  to  any  large 
computer-based  system  whether  it  be  military  or  commercial  and  they  can  be 
easily  tailored  to  fit  a  small  computer  system. 


iii 


INTRODUCTION 


While  much  has  been  written  about  the  design  and  development  of  large 
scale  computer-based  systems,  little  has  been  published  about  the  testing  of 
these  systems  and,  in  particular,  about  the  testing  of  computer  programs 
within  the  context  of  a  system.  Similarly,  while  extensive  techniques  for 
design  control  of  system  hardware  have  been  developed  by  the  Air  Force  over 
the  past  five  years,  technical  control  and  design  verification  procedures  for 
computer  programs  have  not  been  aggressively  investigated.  An  Air  Force 
project  was  established  at  the  Electronic  Systems  Division  (AF  Systems 
Command)  to  rectify  this  situation.  Concepts  resulting  from  this  investiga¬ 
tion  are  presented  and  current  procedures  to  insure  design  integrity  of 
computer  programs  by  technical  reviews  of  the  designer !s  efforts  and  by 
tests  during  program  development  are  summarized. 

UNIFORM  SPECIFICATIONS 


Fundamental  to  Air  Force  management  of  computer  program  design,  develop¬ 
ment,  and  testing  is  the  definition  of  computer  programs  as  deliverable 
contract  end  items  (CEI's)  and  the  adaptation  of  the  Air  Force  uniform 
specification  program  to  these  end  items-^^.  The  uniform  specif ications3, 
both  at  the  system  and  computer  program  contract  end  item  (CPCEl)  level, 
consist  of  two  basic  sections:  Section  3;  performance/design  requirements 
section;  and  Section  4,  the  quality  assurance  or  test  requirements  section. 

It  should  be  realized  that  even  as  early  in  the  design  process  as  the  prepara¬ 
tion  of  the  system  and  end  item  specifications,  the  methods  of  testing  end 
item  performance  against  technical  requirements  should  be  known.  It  is  a 
waste  of  time  and  money  to  specify  technical  requirements  if  a  method  of 
performance  verification  is  not  available  to  evaluate  the  computer  program 
once  it  is  developed.  Generally,  a  one  to  one  relationship  exists  between 
Sections  3  and  4  of  the  specification.  Thus,  each  requirement  of  Section  3 
will  have  an  appropriate  test  requirement  and  test  method  identified  in 
Section  4.  The  specified  requirements  form  the  basis  for  three  formal 
technical  reviews  throughout  the  system/CEI  /  design  and  development.  Con¬ 
currently,  the  specification  test  requirements  are  the  basis  for  subsequent 
test  planning  documentation  and  system  testing  at  both  the  CEI  and  system 


ESD  Exhibit  EST-1,  Configuration  Management  Exhibit  For  Computer  Programs. 
Test  Division  of  the  Technical  Requirements  and  Standards  Office,  Electronic 
Systems  Division,  Air  Force  Systems  Command,  L.G.  Hanscom  Field,  Bedford, 
Massachusetts,  May  1966. 

2 

AFSCM  375-1;  Configuration  Management  During  Definition  and  Acquisition 
Phases .  Andrews  AFB,  Washington,  D.C.  Headquarters,  Air  Force  Systems 
Command,  1  June  1964. 


^Liebowitz,  B.H.  The  Technical  Specification — Key  to  Management  Control  of 
Computer  Programming.  SJCC  Proceedings,  1967* 


performance  levels.  The  relationship  between  the  specifications,  design 
reviews,  and  test  program  is  illustrated  in  Figure  1.  The  lower  blocks  of 
the  figure  identify  the  design  reviews  during  the  system  life  cycle  each  of 
which  will  now  be  discussed  in  more  detail. 

SYSTEM  DESIGN  REVIEW 


The  System  Design^Review  (SDR)  is  held  late  in  the  Definition  Phase*  of 
the  system  life  cycle  .  The  purpose  of  this  first  review  is  to  study  the 
contractor's  system  design  approach.  At  the  SDR  a  critical  examination  is 
performed  to  insure  that  contractor's  design  reflects  a  proper  understanding 
of  all  technical  requirements.  An  analysis  of  contractor  documentation  in 
the  form  of  functional  diagrams,  trade  study  reports,  schematic  diagrams, 
initial  design  specifications,  etc.,  is  conducted.  A  prime  objective  of  the 
SDR  is  to  review  the  allocation  of  functional  requirements  to  the  various 
system  segments  and  CEI’s.  Thus,  for  computer  programs,  the  SDR  must  insure 
that  only  those  system  requirements  that  can  be  realistically  satisfied  by 
computer  programs  have  been  allocated  to  CPCEI's  (i.e.,  operational,  utility, 
diagnostic,  etc.).  Prior  to  the  conduct  of  the  SDR,  trade-off  studies 
concerning  equipments  versus  computer  programs  must  have  been  completed  to 
provide  a  cost  effective  allocation  of  requirements.  Satisfactory  completion 
of  the  SDR  permits  preparation  of  the  Part  I  specifications  ("design  to" 
specifications)  for  all  CPCEI's.  These  specifications  form  the  basis  for  the 
second  technical  review  in  the  design  process. 

PRELIMINARY  DESIGN  REVIEW 


The  Preliminary  Design  Review  (PDR)  is  normally  held  within  60  days  after 
the  start  of  the  Acquisition  Phase.  Concurrently,  the  preliminary  design  of 
the  CPCEI  can  progress  based  upon  the  approved  "design  to"  specifications 
for  the  end  item.  The  purpose  of  the  PDR  is  to  evaluate  the  design  approach 
for  the  end  item,  or  group  of  end  items,  in  light  of  the  overall  system 
requirements;  thus,  the  prime  objective  of  the  PDR  is  achieving  design 
integrity.  A  review  of  the  interfaces  affecting  the  CPCEI  is  an  important 
element  of  a  PDR.  Emphasis  is  placed  on  verification  of  detailed  interfaces 
with  equipment  and  with  other  CPCEI's.  At  the  PDR  the  instruction  set  of 
the  computer  to  be  used  must  be  firmly  established.  The  programming  features 
of  the  computer,  e.g.,  interrupts,  multiprocessing,  time  sharing,  etc.,  must 
be  known.  All  external  data  formats  and  timing  constraints  must  be  identified. 
The  computer  program  storage  requirements  and  data  base  design  are  reviewed 
for  technical  adequacy  at  this  time.  The  structure  of  the  CPCEI  is  also 
reviewed  at  the  PDR.  During  the  initial  design  process  for  a  complex  CPCEI, 


*Phases  as  discussed  here  (i.e.,  Conceptual,  Definition,  Acquisition,  and 
Operational  Phases)  refer  to  the  four  phases  of  the  System  Life  Cycle  as 
defined  in  AFSCM  375-^  System  Program  Management  Procedures. 
k 

Ratynski,  M.V.  The  Air  Force  Computer  Program  Acquisition  Concept.  SJCC 
Proceedings,  19^7^ 
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COMPUTER  PROGRAM  DESIGN  FLOW 
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DEFINITION  PHASE - 4* - ACQUISITION  PHASE 


the  requirements  of  the  Part  I  specification  which  are  function-oriented  are 
allocated  to  computer  program  components  or  modules.  The  relationship  of 
the  components  of  a  typical  CPCEI  to  the  functions  identified  in  a  Part  I 
CPCEI  specification  is  shown  in  Figure  2.  The  allocation  of  functions  to 
computer  program  components  within  the  CPCEI  is  examined  at  the  PDR.  The 
primary  product  of  the  review  at  this  level  is  establishing  the  integrity  of 
the  design  approach,  verifying  compatibility  of  the  design  approach  with  the 
Part  I  specification,  and  verifying  the  functional  interfaces  with  other 
CEI's  in  order  that  detailed  design  of  the  CPCEI  and  its  components  can 
commence. 

CRITICAL  DESIGN  REVIEW 

The  Critical  Design  Review  (CDR)  is  a  formal  technical  review  of  the 
design  of  the  CPCEI  at  the  detailed  flowchart  level.  It  is  accomplished  to 
establish  the  integrity  of  the  computer  program  design  prior  to  coding  and 
testing.  This  does  not  preclude  any  coding  prior  to  the  CDR  required  to 
demonstrate  design  integrity,  such  as  testing  of  algorithms.  In  the  case  of 
a  complex  CPCEI,  as  the  design  of  each  component  proceeds  to  the  detailed 
flowchart  level,  a  CDR  is  held  for  that  component.  In  this  manner,  the  CDR 
is  performed  incrementally  by  computer  program  components,  and  the  reviews 
are  scheduled  to  optimize  the  efficiency  of  the  overall  CDR  for  the  end  item 
as  a  whole.  Due  to  the  varying  complexity  of  the  parallel  design  efforts 
for  CPCEI  components,  it  would  be  unreasonable  to  delay  all  of  the  components 
being  developed  to  hold  one  CDR  for  the  CPCEI. 

At  the  CDR,  the  completed  sections  of  the  Part  II  CPCEI  specification 
(detailed  technical  description)  axe  reviewed  along  with  supporting  analytical 
data,  test  data,  etc.  The  compatibility  of  the  CPCEI  design  with  the  require¬ 
ments  of  the  Part  I  specification  is  established  at  the  CDR.  "Inter”  inter¬ 
faces  with  other  CPCEI ’s  and  "intra"  interfaces  between  computer  program 
components  are  examined.  Design  integrity  is  established  by  review  of 
analytical  and  test  data,  in  the  form  of  logic  designs,  algorithms,  storage 
allocations,  and  associated  methodology.  In  general,  the  primary  product  of 
the  CDR  is  the  establishment  of  the  design  as  the  basis  for  continuation  of 
the  computer  program  development  cycle.  Immediately  following  the  CDR, 
coding  of  individual  components  takes  place  and  the  process  of  checkout  and 
testing  of  the  components  begins. 

COMPUTER  PROGRAM  TESTING 


System  testing  as  defined  by  the  Air  Force  is  divided  into  three  classes 
or  categories  of  testing,  two  of  which,  Category  I  and  Category  11^,  are 


JAFR  80-l4,  Testing/Evaluation  of  Systems,  Subsystems,  and  Equipments. 
Washington,  D.C.  Department  of  the  Air  Force,  14  August  1963 . 
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ALLOCATION  OF  PERFORMANCE  FUNCTIONS 
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(N)  FUNCTIONS 


important  in  development  testing  of  Air  Force  systems  and  will  be  discussed 
here.  Category  I  tests  for  CPCEI' s  are  conducted  by  the  contractor  with 
active  Air  Force  participation.  These  activities,  when  properly  planned  and 
managed,  will  normally  proceed  in  such  a  way  that  testing  and  functional 
demonstrations  of  selected  functions  or  individual  computer  program  components 
can  begin  early  during  acquisition  and  progress  through  successively  higher 
levels  of  assembly  to  the  point  at  which  the  complete  CPCEI  is  subjected  to 
formal  qualification  testing.  Since  the  total  process  is  typically  lengthy 
and  represents  the  major  expense  of  computer  program  acquisition  for  the 
system,  the  test  program  includes  preliminary  qualification  tests  at  appro¬ 
priate  stages  for  formal  review  by  the  Air  Force.  While  the  tests  are 
preliminary  in  nature  (they  do  not  imply  acceptance,  or  formal  qualification), 
they  do  serve  the  necessary  purpose  of  providing  check  points  for  monitoring 
the  contractor's  progress  towards  meeting  design  objectives  and  of  verifying 
detailed  performance  characteristics,  which,  because  of  sheer  numbers  and 
complexity,  may  not  be  feasible  to  verify  in  their  entirety  during  formal 
qualification  testing.  Category  II  tests  are  complete  system  tests,  including 
the  qualified  computer  program  end  items,  conducted  by  the  Air  Force  with 
contractor  support  in  as  near  an  operational  configuration  as  is  practicable. 

Computer  program  testing  is  accomplished  in  accordance  with  Air  Force 
approved  test  documentation  as  illustrated  in  Figure  3*  As  previously 
discussed,  test  requirements  and  corresponding  test  methods  (to  the  level  of 
detail  necessary  to  clearly  establish  the  scope  and  accuracy  of  the  methods) 
are  contained  in  Section  3  and  Section  4  of  the  CEI  specification.  The 
Category  I  test  requirements  are  further  amplified  in  the  contractor-prepared 
Category  I  Test  Plan.  This  document  contains  comprehensive  planning  informa¬ 
tion  for  qualification  tests;  complete  with  schedules,  test  methods  and 
criteria,  identification  of  simulated  versus  live  inputs  and  support  require¬ 
ments  for  test  equipment,  facilities,  special  test  computer  programs  and 
personnel.  The  Test  Plan  forms  the  basis  for  Category  I  Test  Procedures 
which  are  also  prepared  by  the  contractor.  These  describe  individual 
Category  I  qualification  tests  in  detailed  terms,  specifying  objectives, 
inputs,  events,  recording/data  reduction  requirements  and  expected  results. 
Actual  test  results  are  reported  in  a  formal  Category  I  Test  Report. 

CATEGORY  I  (QUALIFICATION)  TESTING  OF  COMPUTER  PROGRAMS 

The  Category  I  test  program  verifies  that  the  CPCEI  satisfies  the 
performance/design  requirements  of  the  Part  I  "design  to"  CPCEI  specifica¬ 
tion.  The  test  program  must  be  designed  to  insure  that  all  of  the  functional 
requirements,  as  translated  into  computer  program  components,  are  tested  and 
that  requirements  are  not  lost  in  the  translation.  The  program  is  divided 
into  two  major  classes  of  tests:  Preliminary  Qualification  Tests  (PQT)  and 
Formal  Qualification  Tests  (FQT) .  The  former  are  designed  to  verify  the 
performance  of  individual  components  prior  to  an  integrated  formal  qualifica¬ 
tion  of  the  complete  CPCEI. 
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TEST  DOCUMENTATION 
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Figure 


PRELIMINARY  QUALIFICATION  TESTING 


The  PQT  phase  is  conducted  incrementally  by  components  in  the  same  manner 
as  the  CDR.  Figure  4  depicts  the  relationship  between  CDR  and  the  Category 
I  test  program.  The  crosshatched  blocks  in  Figure  4  indicate  coding  of 
individual  computer  program  components.  The  PQT's  are  modular  and  a  "building 
block"  effect  occurs  as  testing  progresses.  As  each  computer  program  compo¬ 
nent  is  added  and  each  PQT  conducted,  increased  confidence  develops  in  the 
CPCEI  being  tested.  Generally,  parameter  tests  are  conducted  prior  to  and 
in  parallel  with  PQT's^. 

Parameter  tests  are  those  designed  to  prove  that  an  individual  subpro¬ 
gram  satisfies  the  detailed  design  specification,  not  that  the  program 
performs  as  coded.  These  tests  compare  the  actual  operation  of  each  subpro¬ 
gram  against  the  design  specification.  Parameter  testing  usually  requires  a 
utility  system  incorporating  sophisticated  parameter  test  tools,  which  are 
computer  programs  themselves,  allow  more  efficient  testing  because  they 
increase  the  ease  with  which  a  test  can  be  specified,  Implemented,  and 
analyzed.  They  allow  a  programmer  to  easily  input  data,  make  corrections, 
and  record  results.  In  addition  to  the  test  tools,  the  programmer  needs  the 
compiled  or  assembled  program,  the  Part  I  specifications,  the  test  plan,  and 
the  test  procedures. 

In  parameter  tests,  the  programmer  must  input  a  simulated  program  environ¬ 
ment  to  the  computer.  The  environment  should  include  the  broadest  range  of 
anticipated  inputs,  including  some  illegalities.  The  program  is  operated  in 
this  simulated  environment,  and  the  actual  outputs  are  compared  with  the 
expected  outputs.  After  each  test  run,  the  programmer  analyzes  the  results 
and  makes  corrections  to  the  code.  All  corrections  are  verified  by  sub¬ 
mitting  them  to  a  parameter  test  once  again. 

Assembly  testing  verifies  that  the  computer  program  components  in  the 
CEI  interact  according  to  design  specifications.  It  is  conducted  with 
simulated  inputs  in  order  to  minimize  the  effects  of  people  and  equipment 
and  allows  a  broad  range  of  input  conditions  to  be  simulated.  Elaborate 
test  tools,  such  as  input  simulation,  recording,  and  reduction  programs,  are 
required  to  conduct  assembly  tests.  Since  these  programs  take  time  to 
prepare,  test  requirements  such  as  instrumentation  must  be  anticipated.  For 
example,  provision  must  be  made  for  core  storage  to  accommodate  test  control 
and  test  recording  programs  along  with  the  program  being  tested. 

At  the  conclusion  of  Preliminary  Qualification  Testing,  all  of  the 
computer  program  components  will  have  been  integrated  and  tested  and  the 
CPCEI  is  being  readied  for  formal  qualification  and  acceptance. 


Farr,  Leonard  A.  A  Description  of  the  Computer  Program  Implementation 
Process .  System  Development  Corporation,  TM- 102 1/002/00,  25  February  1963- 
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FORMAL  QUALIFICATION  TESTING 


Qualification  testing  of  a  complex  CPCEI  requires  extensive  use  of  simu¬ 
lation  techniques.  The  use  of  these  techniques  is  dictated  by  the  high  cost 
of  providing  overhead  computer  facilities  or  by  the  unavailability  of  new 
computers  undergoing  a  parallel  design  and  development  effort.  Although 
PQT's  will  make  maximum  use  of  simulation  techniques,  the  FQT’s  of  an  opera¬ 
tional  CPCEI  will  require  live  inputs,  live  outputs,  and  operationally- 
configured  equipment.  A  prerequisite,  then,  of  FQT  is  usually  the  installation 
and  checkout  of  the  CPCEI  in  an  operationally-configured  system  at  the  Category 
II  test  site.  The  exception  would  be  in  the  case  of  a  support  CPCEI  such  as  a 
compiler  that  would  require  live  inputs,  e.g.,  radar  data,  and  could  be  fully 
qualified  at  the  contractor's  facility.  To  provide  reliable  data  during  FQT, 
the  CPCEI  installation  requires  fully  installed  and  checked  out  equipment 
CEITs.  The  first  opportunity  for  FQT  will  normally  occur  at  the  Category  II 
test  site  after  equipment  CEI's  that  have  successfully  passed  First  Article 
Configuration  Inspection  have  been  installed  and  checked  out  and  an  opera¬ 
tionally-configured  system  exists.  FQT  is  conducted  subsequent  to  installa¬ 
tion  and  checkout  of  the  CPCEI.  The  conclusion  of  FQT  signals  the  end  of  the 
Category  I  test  program.  The  CPCEI  will  have  been  fully  qualified  and  all 
of  the  requirements  of  the  Part  I  specification  should  have  been  satisfied 
except  for  those  requirements  of  the  Part  I  specification  that  can  only  be 
demonstrated  during  a  Category  II  system  test.  After  successfully  passing 
this  phase  of  testing,  the  CPCEI  is  fully  integrated  into  the  system  and  is 
ready  for  system  testing. 

FIRST  ARTICLE  CONFIGURATION  INSPECTION 

With  CPCEI  design  and  testing  essentially  completed,  Part  II  of  the  CPCEI 
specification  is  available  for  review.  The  Part  II  specification  provides  a 
complete  and  detailed  technical  description  of  the  CPCEI  "as  built,"  including 
all  changes  resulting  from  prior  testing.  It  will  accompany  the  CPCEI  to  each 
installation  or  site  and  functions  as  the  primary  document  for  "maintenance" 
of  the  CPCEI.  Consequently,  the  technical  accuracy  and  completeness  of  the 
Part  II  specification  must  be  determined  prior  to  its  acceptance  by  the  Air 
Force.  The  First  Article  Configuration  Inspection  (FACl)  provides  the 
vehicle  for  the  required  review;  thus,  it  is  an  audit  of  the  Part  II  CPCEI 
specification  and  the  CPCEI  as  delivered.  The  primary  product  of  the  FACI 
is  the  formal  acceptance  by  the  Air  Force  of:  (l)  the  CPCEI  specification 
(Part  II)  as  an  audited  and  approved  document;  and  (2)  the  first  unit  of  the 
CPCEI.  Air  Force  acceptance  of  the  CPCEI  is  based  on  the  successful  comple¬ 
tion  of  the  Category  I  Test  Program  and  the  FACI,  but  it  does  not  relieve 
the  contractor  from  meeting  the  requirements  in  the  system  specification. 
Subsequent  to  FACI,  the  configuration^  of  the  CPCEI  is  essentially  controlled 
at  the  machine  instruction  level  so  that  the  exact  configuration  of  the  CPCEI 
is  available  for  Category  II  system  testing. 
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CATEGORY  II  SYSTEM  TESTING 


After  acceptance  of  the  CPCEI,  the  Air  Force  conducts  an  extensive 
Category  II  system  test  program  with  the  objective  of  demonstrating  that  the 
total  system  meets  the  performance/design  requirements  specified  in  the  System 
Specification.  Insofar  as  the  computer  programs  are  concerned,  Category  II 
testing  will  verify  the  CPCEITs  compatibility  with  the  system  elements  and 
its  integrated  performance  in  meeting  system  requirements  in  the  live  environ¬ 
ment,  with  operational  communications,  personnel,  etc.  Residual  design  and 
coding  errors  discovered  in  this  phase  of  testing  are  corrected  prior  to  the 
system  becoming  operational. 

SUMMARY 


The  techniques  for  design  reviews  and  testing  presented  in  this  paper 
provide  a  means  of  insuring  the  design  integrity  of  computer  programs  during 
the  lengthy  design  and  development  cycle.  It  provides  the  Air  Force  with 
technical  control,  at  discrete  phase  points,  which  was  not  before  available. 
To  provide  this  control,  existing  Air  Force  Systems  Command  management 
techniques  were  assessed  and  adapted  to  computer  programs.  No  attempt  has 
been  made  to  equate  computer  programs  with  equipments;  rather,  the  require¬ 
ment  for  similar  technical  controls  has  been  recognized  with  due  considera¬ 
tion  for  the  inherent  differences  between  computer  programs  and  equipment. 
While  these  techniques  were  developed  for  computer  programs  within  the 
context  of  large  computer-based  systems,  they  are  and  have  been  readily 
adaptable  to  small  individual  computer  program  procurements.  More  detailed 
information  on  requirements  and  procedures  are  included  in  ESD  Exhibit  EST-1 
and  ESD  Exhibit  EST-2,  published  by  the  Electronic  Systems  Division. 

Though  the  above  techniques  have  been  used  on  contracts  at  ESD,  none  of 
the  programs  have  progressed  through  the  complete  cycle.  The  limited  experi¬ 
ence  to  date  indicates  that  the  techniques  are  feasible,  they  do  provide 
vitally  needed  Air  Force  technical  control  and  visibility  and,  in  turn,  they 
have  been  useful  to  the  contractors  as  a  formal  management  scheme  and  a 
means  for  mutual  understanding  and  problem  resolution. 
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